ARTICLE
ABSTRACT
In this article the problem of the measurement of agreement among experts is discussed. There are several tools to verify whether opinions expressed by an ordinal scale are reliable. If the objects are ranked, one of correlation coefficients is chosen. At the beginning, the case of m > 2 experts ranking n objects is presented; first, using untied ranks and second, tied ranks. To this end Kendall’s coefficient of concordance is applied. The method of examination of the statistical significance is also presented. Additionally the issue of the interpretation of the concordance coefficient in reference to the average of Spearman’s r coefficients is raised. The paper provides an example of using concordance coefficient when ranks are tied and discusses limitations of using rank correlation methods as a tool for evaluating the degree of agreement among experts’ opinions.
KEYWORDS
experts’ opinions, ranks, tied ranks, mid-rank method, rank correlation, concordance coefficient
REFERENCES
[1] Brzeziński J., Maruszewski T., [1978], Metoda sędziów kompetentnych i jej zastosowanie w badaniach pedagogicznych, „Kwartalnik Pedagogiczny”, Nr 1(87).
[2] Damodaran A., [2009], Ryzyko strategiczne, Wydawnictwa Akademickie i Profesjonalne, Warszawa.
[3] Day S.G., Schoemaker P.J.H., [2006], Peripheral Vision. Detecting Weak Signals That Will Make or Break Your Company, Harvard Business School Press, Boston.
[4] Doskonalenie struktur organizacyjnych przedsiębiorstw w gospodarce opartej na wiedzy, [2009], pod red. A. Stabryły, Wydawnictwo C.H. Beck, Warszawa.
[5] Góralski A., [1976], Metody opisu i wnioskowania statystycznego w psychologii, PWN, Warszawa.
[6] Grouard B., Meston F., [1997], Kierowanie zmianami w przedsiębiorstwie, Poltex, Warszawa.
[7] Kendall M., [1962], Rank correlation methods, Charles Griffin & Company, London.
[8] Sharpe N.R., Veaux De R.D., Valleman P.F., [2009], Business Statistics, Pearson, New York.
[9] Siegel S., [1956], Nonparametric statistics for the behavioral sciences, McGraw-Hill Company Inc., New York.
[10] Simons R., [2005], Czy wiesz jak duże ryzyko ukryte jest w twojej firmie?, „Harvard Business Review Polska”, kwiecień.
[11] Stabryła A., [2000], Zarządzanie strategiczne w teorii i praktyce firmy, Wydawnictwo Naukowe PWN, Warszawa
[12] Steczkowski J, Zeliaś A., [1997], Metody statystyczne w badaniach zjawisk jakościowych, Wydawnictwo Akademii Ekonomicznej w Krakowie, Kraków.
[13] The Delphi Method. Techniques and Applications, [1975], pod red. H. Linstonea I M. Turoffa, Addiosn-Wesley Publishing Company Inc. London.
[14] Winer B.J., [1962], Statistical principles in experimental design, McGraw-Hill Company Inc., New York.
